{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "joined-exhibit",
   "metadata": {},
   "source": [
    "# Insights\n",
    "This section introduces the insights supported by dataprep"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "unique-flight",
   "metadata": {},
   "outputs": [],
   "source": [
    "%reload_ext autoreload\n",
    "%autoreload 2\n",
    "from dataprep.datasets import load_dataset\n",
    "from dataprep.eda import plot, plot_correlation, plot_missing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "important-recording",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = load_dataset(\"titanic\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "advanced-equilibrium",
   "metadata": {},
   "outputs": [],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "relative-buffalo",
   "metadata": {},
   "source": [
    "## Where is the insights\n",
    "\n",
    "We give an example in the following: 1. Click the bottom \"Show Stats and Insights\" 2. We could see the insights provided by Dataprep.\n",
    "\n",
    "![avatar]()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bearing-northeast",
   "metadata": {},
   "source": [
    "Then we could find the insights that are provided by Dataprep in this section.\n",
    "\n",
    "![avatar]()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "engaged-prince",
   "metadata": {},
   "source": [
    "If we use `plot(df, col)` function, we have to click the following buttom:\n",
    "\n",
    "![avatar]()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "paperback-myanmar",
   "metadata": {},
   "source": [
    "Then we could see the following insights:\n",
    "\n",
    "![avatar]()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "secure-assignment",
   "metadata": {},
   "source": [
    "## The insights provided by plot(df)\n",
    "Here we give an example to show insights that could be provided by plot(df).\n",
    "\n",
    "| insights | applied plots | type | threshold | discription |\n",
    "|  :----  | :----  | :----| :---- | :---- |\n",
    "| Duplicates  | Overview | int | 1 | Warn if the percent of duplicated values is above this threshold. |\n",
    "| Negatives | Overview | int | 1 | Warn if the percent of megatives is above this threshold. |\n",
    "| Similar_distribution  | Overview | float | 0.05 | The significance level for Kolmogorov–Smirnov test. |\n",
    "| Uniform  | Histogram | float | 0.999 | The p-value threshold for chi-square test. |\n",
    "| Missing  | Histogram | int | 1 | Warn if the percent of missing values is above this threshold. |\n",
    "| Skewed  | Histogram | float | 1e-5 | The p-value for the scipy.skewtest which test whether the skew is different from the normal distributionin. |\n",
    "| Infinity  | Histogram | int | 1 | Warn if the percent of infinites is above this threshold. |\n",
    "| Zeros  | Histogram | int | 5 | It shows some columns that have zero values larger than the threshold. |\n",
    "| Normal | Histogram | float | 0.99 | The p-value threshold for normal test, it is based on D’Agostino and Pearson’s test that combines skew and kurtosis to produce an omnibus test of normality. |\n",
    "| High Cardinality  | Bar Chart | int | 50 | The threshold for unique values count, count larger than threshold yields high cardinality. |\n",
    "| Constant | Bar Chart | int | 1 | The threshold for unique values count, count equals to threshold yields constant value. |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "addressed-boundary",
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "plot(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fatal-recovery",
   "metadata": {},
   "source": [
    "## The insights provided by plot(df, col) when col is a continues column\n",
    "\n",
    "Here we give an example to show the insights could be yielded by plot(df, x), when x is a continues column.\n",
    "\n",
    "| insights | applied plots | type | threshold | discription |\n",
    "|  :----  | :----  | :----| :---- | :---- |\n",
    "| Infinity  | Stats | int | 1 | Warn if the percent of infinites is above this threshold. |\n",
    "| Missing  | Stats | int | 1 | Warn if the percent of missing values is above this threshold. |\n",
    "| Negatives  | Stats | int | 1 | Warn if the percent of megatives is above this threshold. |\n",
    "| Zeros  | Stats | int | 5 | Warn if the percent of zeros is above this threshold. |\n",
    "| Normal  | Histogram, Normal Q-Q Plot | float | 0.99 | The p-value threshold for normal test, it is based on D’Agostino and Pearson’s test that combines skew and kurtosis to produce an omnibus test of normality. |\n",
    "| Uniform  | Histogram | float | 0.999 | The p-value threshold for chi-square test.|\n",
    "| Skewed  | Histogram | float | 1e-5 | The p-value for the scipy.skewtest which test whether the skew is different from the normal distributionin. |\n",
    "| Outliers  | Box Plot | int | 0 | It shows how many outliers a column has.|"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "generic-camel",
   "metadata": {},
   "outputs": [],
   "source": [
    "plot(df, \"Age\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "streaming-neutral",
   "metadata": {},
   "source": [
    "## The insights provided by plot(df, col) when col is a nominal column\n",
    "\n",
    "Here we give an example to show the insights could be presented by plot(df, col), when col is a nominal column.\n",
    "\n",
    "| insights | applied plots | type | threshold | discription |\n",
    "|  :----  | :----  | :----| :---- | :---- |\n",
    "| Constant | Stats | int | 1 | The threshold for unique values count, count equals to threshold yields constant value. |\n",
    "| High_cardinality | Stats | int | 50 | The threshold for unique values count, count larger than threshold yields high cardinality. |\n",
    "| Missing | Stats | int | 1 | Warn if the percent of missing values is above this threshold. |\n",
    "| Uniform | Bar Chart | float | 0.999 | The p-value threshold for chi-square test. |\n",
    "| Outstanding_no1  | Bar Chart | float | 1.5 | It measures the ratio of the largest category count to the second-largest category count. |\n",
    "| Attribution  | Pie Chart | float | 0.5 | It measures the percentage of the top 2 categories. |\n",
    "| High_word_cardinality | Word Cloud | int | 1000 | The threshold for the high word cardinality insight, which measures the number of words of that cateogory. |\n",
    "| Outstanding_no1_word | Word Cloud | int | 0 | The threshold for the outstanding no1 word threshold, which measures the ratio of the most frequent word count to the second most frequent word count. |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "optical-tooth",
   "metadata": {},
   "outputs": [],
   "source": [
    "plot(df, \"Sex\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}